refactor(performance): discover and trigger reference-based syncs asynchronously#930
Open
rohan-chaturvedi wants to merge 21 commits into
Open
refactor(performance): discover and trigger reference-based syncs asynchronously#930rohan-chaturvedi wants to merge 21 commits into
rohan-chaturvedi wants to merge 21 commits into
Conversation
… for cross-references
… secret references
Signed-off-by: rohan <rohan.chaturvedi@protonmail.com>
Signed-off-by: Rohan Chaturvedi <rohan.chaturvedi@protonmail.com>
Signed-off-by: Rohan Chaturvedi <rohan.chaturvedi@protonmail.com>
…override references
…ferences # Conflicts: # backend/api/models.py # backend/api/tasks/syncing.py # backend/tests/tasks/test_syncing.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
#827 (now on
main) added detection that, on every secret write, finds the other environments whose secrets reference the changed env and triggers their syncs. That detection runs insideEnvironment.save()— synchronously on the request path — and decrypts + walks the org's entire reference graph, so its cost is paid on every secret write and scales with the org's synced footprint.Change
Move the cross-reference detection off the request path into an RQ job:
Environment.save()still triggers the changed env's own syncs synchronously (so theirqueuedstatus shows immediately).detect_and_trigger_referencing_syncs.delay(str(self.id))and runs in a worker.No change to what gets synced — only when the cross-reference detection runs (shortly after the write instead of within it). Bulk writes already batch to one detection per affected env.
Deploy note
This adds a new RQ job, so deploy/restart the workers with (or before) the backend so they can run it. One-time, standard hygiene for any new task; the changed env's own syncs still fire synchronously, so only the cross-reference fan-out is briefly deferred during the rollout window.